Systems and methods for presenting location related information

ABSTRACT

Systems and methods for presenting location related information after a user arrives at a place. In an aspect, when a user gazes at a display of a standby device, information presentation begins. In other aspects, when a user shakes a device or says certain words to a device, and then gazes at it, a presentation starts.

CROSS REFERENCE TO RELATED APPLICATION

This is a continuation-in-part of U.S. application Ser. No. 15/397,726,filed Jan. 3, 2017, which is a continuation-in-part of U.S. applicationSer. No. 14/525,194, filed Oct. 27, 2014, now U.S. Pat. No. 9,619,022,granted Apr. 11, 2017.

FEDERALLY SPONSORED RESEARCH

Not applicable

SEQUENCE LISTING OR PROGRAM

Not applicable

BACKGROUND Field of Invention

This invention relates to presenting information, more particularly topresenting information utilizing gaze detection.

Description of Prior Art

Many portable electronic devices have become ubiquitous, as anindispensible part of our daily life. Examples include smartphones,tablet computers, and potentially, smart watches or other wearablegadgets. These devices, especially smartphones, may be used to transmitto users and then present information such as advertisement prepared forconsumers, notice and info for event attendees, class messages forstudents, or flight info for passengers. But many a time, it is not easyto acquire contact info of people involved and to figure out when topresent. For instance, most ads are delivered to peopleindiscriminately, blindly, and without specific consideration on timing,which compromises the effectiveness of ads.

To make ads more relevant and acceptable, location-based advertising hasbeen advocated. For instance, people visiting a store have a betterchance to become a customer than people elsewhere. So a store managermay be more interested in sending ads to people present at the storethan people at home. The same is true for delivery of information otherthan advertisements. For example, event attendees are more willing toread event material when they are in there, students are more likely toread class messages when at school, and passengers are more eager tolearn flight and gate status when at the airport. Moreover, it'srelatively straightforward to send location related information, sincedevices on the scene are the obvious target, and it may start sendingmessages right after users arrive at a location or come near a location.As a result, it's likely that the right info is sent to the right peoplein the right place at the right time. But then, the next issue may behow to present it in such a way that it is easy, simple, and convenientfor a user to access. If relevant info is transmitted via email, amethod used quite often nowadays, people may have to go through severalsteps to log in an email account, open a mail, and then take a look atit. If viewing info requires an app, people have to find the app amongother apps installed at a device and then launch it. Either way, it isnot convenient enough to look for info transmitted from a network orservice to a device. On the other hand, if a device is on, and infocontent pops up by itself, it may become annoying; and if a device is instandby mode with a dark screen, it is inappropriate to lighten up itsdisplay to show any content without user consent. Thus presenting infoon a device automatically has its own limitations.

Therefore, there exists a need to present location related informationin a simple, easy, and convenient way.

OBJECTS AND ADVANTAGES

Accordingly, several main objects and advantages of the presentinvention are:

-   -   a). to provide improved methods and systems to present location        related information at a device;    -   b). to provide such methods and systems which start a        presentation when a user gazes at it;    -   c). to provide such methods and systems which start a        presentation when a user shakes it and then gazes at it;    -   d). to provide such methods and systems which start a        presentation when a user says certain words to it and then gazes        at it;    -   e). to provide such methods and systems which sort location        related information by device pointing direction; and    -   f). to provide such methods and systems which make info access        easy, simple, and convenient.

Further objects and advantages will become apparent from a considerationof the drawings and ensuing description.

SUMMARY

In accordance with the present invention, methods and systems areproposed to present location related information. After a user arrivesat a place, the user may just look at a device screen to start an infopresentation by gaze. The user may also shake a device to trigger gazedetection, and then watch it to bring out a presentation by gaze. Inaddition, the user may speak to a device and then gaze at it to invoke apresentation. The method makes it easy and convenient for a user to getinformation. Moreover, information presented may be sorted by a device'spointing direction, which may reduce contents on a screen and make iteasy to view.

DRAWING FIGURES

FIG. 1 is an exemplary block diagram describing one embodiment inaccordance with the present invention.

FIG. 2 is an exemplary flow diagram showing one embodiment of presentinglocation related information in accordance with the present invention.

FIG. 3 is an exemplary flow diagram showing another embodiment ofpresenting location-based information in accordance with the presentinvention.

FIG. 4 uses graphic diagrams to show yet another embodiment ofinformation presentation in accordance with the present invention.

REFERENCE NUMERALS IN DRAWINGS

-   -   10 Sensor    -   12 Device    -   14 Processor    -   16 Computer Readable Medium    -   18 Sensor    -   20 Sensor    -   22 Sensor    -   36 Eye    -   38 Smartphone    -   100, 102, 104, 106, 108, 110, 112, 114, 116, 118, 120, 122, 124,        and 126 are exemplary steps.

DETAILED DESCRIPTION

FIG. 1 is an illustrative block diagram of one embodiment according tothe present invention. A device 12 may represent an electronic device,including but not limited to mobile phone, smart phone, smart watch,wearable device, handheld computer, tablet computer, and the like.Device 12 may include a processor 14 and computer readable medium 16.Processor 14 may mean one or more processor chips or systems. Medium 16may include a memory hierarchy built by one or more memory chips orstorage modules like RAM, ROM, FLASH, magnetic, optical and/or thermalstorage devices. Processor 14 may run programs or sets of executableinstructions stored in medium 16 for performing various functions andtasks, e.g., surfing on the Internet, playing video or music, gaming,electronic payment, social networking, sending and receiving emails,messages, files, and data, executing other applications, etc. Device 12may also include input, output, and communication components, which maybe individual modules or integrated with processor 14. The communicationcomponents may connect the device to another device or a communicationnetwork. Usually, Device 12 may have a display (not shown in FIG. 1 forbrevity reason) and a graphical user interface (GUI). A display may haveliquid crystal display (LCD) screen, organic light emitting diode (OLED)screen (including active matrix OLED (AMOLED) screen), or LED screen. Ascreen surface may be sensitive to touches, i.e., sensitive to hapticand/or tactile contact with a user, especially in the case of smartphone, smart watch, and tablet computer. A touch screen may be used as aconvenient tool for user to enter input and interact with a system.Furthermore, device 12 may also have a voice recognition component forreceiving verbal command or audio input from a user.

A communication network which device 12 may be connected to may cover arange of entities such as the Internet or the World Wide Web, a localarea network (LAN), a wide area network (WAN), a metropolitan areanetwork (MAN), a telephone network, an intranet, wireless, and othertypes of networks. Device 12 may be connected to a network by variouswired, wireless, optical, infrared, ultrasonic or other communicationmeans.

Device 12 may also include a sensor 10 which tracks the eye movement orgazing direction of user using mature eye-tracking or gaze detectiontechnologies. The sensor may be arranged on the top surface of device,or close to a display screen, and may be designed to have imagingcapability. With imaging functions, a system may recognize whetherusers' eye is in such a position that the eye sight falls on the body ofdevice 12 using certain algorithm, in other words, sensor 10 may beemployed to determine whether a user is looking at the body or thescreen of a device. Once it senses that a user is gazing or looking at agiven target, it may record the starting time, and then the total gazingor watching time. Only when the gazing or watching time exceeds certainvalue, for instance a few seconds, it may be declared that a user isgazing or looking at a target. As a consequence, a very brief look maybe too short to qualify as a gazing or watching act. In the followingsections, it is assumed the total gazing time of each case satisfies aminimum value requirement when it is said a gazing act is detected.

Sensor 10 may be built using mature imaging technologies, such as cameramodules used in almost every smartphone, and an image of user's eye maybe analyzed with mature algorithm to decide which direction a user islooking at. Both visible and infrared light may be employed for eyetracking. In the latter case, an infrared light source may be arrangedto provide a probing beam. In addition, sensor 10 may also employ othersuitable technologies which are capable and affordable other than theeye-analysis scheme discussed to determine a gazing or watchingdirection of a user. In some applications, when the accuracy of gazingdirection is not critical, such as when a gaze target is a screen, not asmall area of the screen, a watching direction may be obtained viaanalyzing facial pictures of a user.

Device 12 may also include a sensor 20 which functions as a proximitydetector, which is well known in the art and well developed too. Sensor20 may be used to detector an object outside of the device and may havemultiple sensing units. It may include a camera-like system to obtainvisible images or infrared images and then recognize any movementthrough image analysis over a period of time. It may also havecapability to sense whether device 12 is close to a user's body orwhether it is held by a hand. Detection result may be used to determinean environment where a user is in, or the intention of a user. Forinstance, a user may want to look at a device anytime when he is holdingit on hand.

Moreover, device 12 may contain a sensor 18 to detect its own movementby sensing acceleration, deceleration, and rotation, which may bemeasured by accelerometers and gyroscopes. Accelerometers and gyroscopesare already mass produced using semiconductor technologies. They arewidely used in smartphones and other personal gadgets. Using measurementdata obtained by sensor 18, it can be determined whether device 12 ismoved to the left, right, forward, or backward, and at what speed,whether it is rotated clockwise or anticlockwise along which axis, andwhether it is tilted to the left, right, forward, or backward. The datamay also be used to detect whether a device is moved back and forth as aresult of shaking. In some embodiments in the following, device shaking,as a user input, is one state to be detected. Word “shake” or “shaking”may be interpreted here as moving a device horizontally or vertically,rotating along any axis, or any other patterns of back and forthmovement. Furthermore, sensor 18 may be used to detect vibration ofdevice 12. Thus, knocking or tapping on a device body may be utilized asa user input too, because it generates detectable vibration signals.

Inside device 12, output signals of sensors and detectors aretransmitted to processor 14, which, employed with certain algorithm, mayprocess the data and produce subsequent command instructions accordingto certain programs or applications. The instructions may includepresenting location related info on a screen.

In addition, device 12 may carry a positioning sensor (not shown in FIG.1 for brevity) and a magnetic sensor 22 as an electronic compass. Apositioning sensor may be a global positioning system (GPS), whichenables a device to get its own location info. Device position may alsobe obtained using wireless triangulation methods, or a system usingother suitable technologies, while both may be performed by a serviceprovider or service facility. Sensor 22 measures the earth magneticfield along at least two orthogonal axes X and Y. It may be used todetermine device orientation, such as which direction a device ispointing at, assuming the device is placed in a horizontal or verticalposition. When a device's location is known, service center may send tothe device location-based information, i.e., info related to thelocation or nearby places. In the case of location-based advertising, auser may receive commercials after he or she is at a business or closeto a business. On the other hand, when the pointing direction of deviceis known, space around a user may be divided into sections. With theknowledge of device's location and pointing direction, a segment of maparea which corresponds to where a device is pointing at may begenerated. The segment may match user's interest, and thus informationfrom this segment may be more relevant than info from other areas.Meanwhile, sorting by segment may make information easier to view forusers, since contents presented on screen are reduced.

FIG. 2 is a schematic flow diagram showing one embodiment of presentinglocation related information. Take smartphone for example. Assume asmartphone is in standby mode in Step 100. When a user with the phoneenters Location A, a system sensor may detect it in Step 102. Forinstance, when a phone arrives at a place, a service provider may senseit or a local sensor may detect it using mature positioningtechnologies. Assume there is information available which is related toLocation A. In Step 104, a location-based signal is transmitted to thephone and the phone receives it. The signal may come from a remotecenter or a nearby facility. Once the phone gets the signal, it startssensing user's gaze direction. When not triggered, gaze detectionfunction may be in off state to conserve power. In Step 106, the usergazes at the phone screen, which may be sensed by a gaze sensor likesensor 10 of FIG. 1. Here user's gaze act may work as user's approvalfor presenting information. In Step 108, the phone displays contentsrelated to Location A.

After arriving at a location, a user may become more likely to viewinformation related to the place. The user just needs to look at phonescreen, info would appear automatically. The info presentation processis easy, simple and convenient. It may be used by teacher to distributeclass notes, which may be accessed by students at one classroom only, bystore manager to send advertisement to people at or close to his or herstore only, or by organizer to send on-site event participants info onthe event. Usually for indoor or some urban environment, positioningmethods other than GPS are used, since GPS requires a clear view of thesky or clear line of sight for four GPS satellites.

The scheme described in FIG. 2 provides a simple and practical way toarrange location related information. But when lot of such informationis available, it makes things a little complicated. For instance, in ashopping mall area, there may be many stores and shops around. As aconsequence, a user may find it time consuming to get needed info. Thusa quick and convenient sorting method is desirable. For this issue,following discussion gives a solution.

FIG. 3 shows another schematic flow diagram of presenting locationrelated information. Assume a device is standby and is detected at aplace in Step 110. Next in Step 112, the device receives a signal whichcontains location related information through wireless technologies.Then, a gaze sensor is activated and begins to sense the gaze directionof a user. The gaze sensor may be arranged always on if powerconservation is not an issue and the user consents. In Step 114, thegaze sensor detects whether the user looks at the device. If the userlooks elsewhere, the device may remain its standby state in Step 116.When the user ends the standby state later on, a temporary icon mayappear on screen. The icon may represent information related to thelocation. Once the icon is tapped or clicked, location related info maybe presented. A temporary icon may also be generated on screen for lateruse when a user is busy engaging with an app at the moment of receivinglocation related information. Such icon provides another opportunity topresent temporarily stored location related contents. Back to thefigure, if it is detected that the user looks at the device for a givenperiod of time, the device may start to detect its orientation using amagnetometer component like sensor 22 of FIG. 1, as shown in Step 118.In the meantime, the device may acquire its position status, i.e., itslocation. Location data may be obtained via the device's own sensor oran outside sensing system. Once info of location and orientation isknown, the device may start presentation of related information in Step120. The related information is of info associated with the pointingdirection of the device. For instance, with the knowledge of locationand orientation and certain algorithm, a device may provide a list ofbusinesses which are located between its place and somewhere far awayalong its pointing direction. The list of businesses may be in a textfile or shown on a map segment. A map segment is part of a map with anelongated shape along a device pointing direction. A map segment may beobtained by cutting off some parts of a map and leaving only anelongated segment. Thus a pointing act may be used as a sorting tool,and a device may be designed to show information related to or around apointing direction only. Besides businesses and organizational entities,pointing direction of device may also be used to get info of products.For instance, a user may point a device at one section of a store to getprearranged info of that area, such as coupons and items on sale in thatdirection.

A device may be in a horizontal position, or vertical position. Takesmartphone for instance. If a phone is in horizontal position, with itsdisplay screen being horizontal and parallel to the ground, a pointingdirection is what its front end points outwards in a horizontal plane.For a phone in vertical position, a pointing direction is what its backpoints at or its camera points at, which is the opposite direction ofwhat its screen faces.

As orientation data may be obtained fast through an electronic compass,a pointing act may lead to real-time info scanning. In Step 122, deviceorientation is measured again. If there is no change, contents ondisplay may remain in Step 124. If there is a change, meaning the deviceis rotated to point at a new direction, another set of contents may bepresented in response in Step 126. For example, when a user rotates asmartphone horizontally along a vertical axis, it may work like scanningwith a probing beam. It may be designed such that during scanning, onlyinformation related to a business which is straight ahead will show upon screen. Thus a user may slowly rotate a device, like a smartphone, toview info at each direction, or point a device at a selected business toaccess info of that business directly.

FIG. 4 uses graphic diagrams to show another embodiment of presentinglocation related information. Again, a smartphone is used in a retailsetting. It starts with Step 1 when a positioning sensor finds asmartphone 38 at store A. The phone is in standby mode and has a darkscreen. A service facility sends the phone a signal, and the phonereceives location related information. Unlike the previous embodiment,gaze sensor of the device is not triggered by the location-based signal,but by user's physical act like shaking or tapping the device. In Step2, the user shakes phone 38, which is picked up by the phoneimmediately, e.g., within seconds. Then the control system of phone 38,like processor 14 of FIG. 1, sends a signal to the gaze sensor. The gazesensor starts sensing the user to determine whether he or she looks atthe phone screen. If it is detected that user's eye 36 is on the phonescreen for a predetermined period of time in Step 3, the device maybegin presenting store advertisements and coupons in Step 4.

In above discussions, a user needs to do two things, shaking a phonelightly and watching its screen briefly, and then certain info will bedisplayed. The scheme brings several merits. A user may have morecontrol over what time to show location related information. It mayreduce chances of showing unwanted info by an accidental gaze at adevice. In addition, as a shaking act reflects user's desire for certaincontent, it may help satisfy a user and help content owner likemerchants in the meantime.

Furthermore, a user may speak to a device to turn on a gaze sensor usingvoice recognition techniques. For instance, a user may say to a device“Start” and then look at it to invoke a presentation. Benefits of usinggaze detection and voice recognition together include precision,convenience, multiple choices, and complex instructions. Without gazedetection, multiple devices may react to a voice command and causeconfusion. Without voice recognition, gazing may invoke a single andoften simple task only, which may limit applications.

When voice recognition and gaze detection are used together, twoscenarios may be created: A user may say certain word or words and thenlook at a device or look at a device and then say certain word or words.The two actions, i.e., speaking and gazing, in both scenarios may bearranged to cause a device to carry out one or more tasks. Asaforementioned, when it is detected that a user looks at or gazes at adevice, it means the user looks at or gazes at it for at least a giventime. The tasks may include presenting certain contents, turning on adevice from a standby or power-off state, switching from one workingstate to another one, implementing one or more tasks specified in avoice input, and performing other given tasks. For brevity purpose, onlyone or two tasks are cited when discussing voice-related examples below,where other tasks may be applied without mentioning. Contents presentedusing or at a device may be related to a location, scheduled by a user,arranged by a remote facility or service center, or specified in a voiceinput. The contents may have video, audio, or other format and may besubscribed with fees or sponsored by an entity. A device may presentcontents using a display, a speaker, or other output component.Initially, the device may be at a standby, sleeping, power-off, orpower-on state. In some applications, whether or not a user gazes at adevice may be detected. In other applications, whether or not a usergazes at a device's display, speaker, or other output component may bedetected. For brevity reasons, only the former case, i.e., gazing at adevice, is mentioned in discussions below.

In the first scenario, a voice recognition system is on and monitoring auser's voice message from the beginning. For instance, the system may bearranged in an operational mode to collect and analyze a user's voicemessage continuously. After the system receives a voice input, itanalyzes and interprets the input using certain algorithm and ascertainswhether the input matches or contains one of prearranged voice commands.A single word or sentence such as “Start”, “Turn on”, a program name, ora device name may mean a command to start a presentation or turn on adevice. Once it is detected that a user issues a voice command, theuser's gaze direction is checked. A gaze sensor may be in a workingstate all the time. Alternatively, the gaze sensor may also be triggeredto wake up from a sleeping or standby state by a signal which may betriggered by the voice recognition system after the system receives aninput. When it is concluded that a user gazes at a device within a givenshort time period, like five to ten seconds, after a voice command isreceived, the command is implemented at the device. If a device cannotascertain that a user gazes at it, the device may ignore a voice commandwhich it received a short while ago. The gaze requirement enablestargeting a device with precision, which may be especially useful whenmultiple devices which all have voice recognition capabilities arepresent.

In the second scenario, a gaze sensor is on and monitors a user's gazedirection continuously. A voice recognition system may remain active andready to take a voice input all the time. As another option, a voicerecognition system may be on standby and only wake up when a gazing acthappens. For instance, after it is detected that a user gazes at adirection towards a device, a signal may be generated to turn on a voicerecognition system at the device and optionally, the device may turn ona lighted sign with a word like “Ready”. The sign may work as aninvitation to ask for voice instructions from a user. As long as a userlooks at the device, the sign may stay lighted there. When it isdetermined that a user gives a voice command while still looking at thedevice or within a given time period, say five to ten seconds, after theuser finishes a gazing act, the voice command is carried out at thedevice. If a user gives a voice command without looking at acorresponding device, the voice command may not take effect. Again,gazing and voice command are used together to target a device withprecision and initiate a task at the device.

When both a gaze sensor and a voice recognition system are turned onfrom the beginning, a method may be arranged where either a gazing actor a voice input act may happen first. For instance, it may beconfigured that if a user says a command and then gazes at a devicewithin a given time, the command may be implemented at the device; if auser says a command and gazes a device at the same time, the command maybe implemented at the device; if a user gazes at a device and then saysa command while still gazing at the device, the command may beimplemented at the device; and if a user gazes at a device and then saysa command within a given time after the gazing act ends, the command maybe implemented at the device. In other words, assume that a user gazesat a device during a first time period from time-A1 to time-A2 andissues a voice command during a second time period from time-B1 totime-B2. The device may be arranged to implement the command if the twotime periods overlap either fully or partially or a gap value betweenthe two time periods along a timeline is smaller than a given value, sayfive to ten seconds, where it doesn't matter which period happens first.For instance, when time-B1 is later than time-A1 and time-B2 is earlierthan time-A2, the two time periods overlap fully. When time-B1 is laterthan time-A1 but earlier than time-A2 and time-B2 is later than time-A2,the time periods overlap partially. When the two time periods don'toverlap, time interval between time-A2 and time-B1 or between timetime-B2 and time-A1 is the gap value. It is seen that above descriptionsusing time periods apply to cases where a gaze sensor or voicerecognition system is triggered by a user's verbal or gazing action.

When multiple devices are involved, two methods may be designed. Assumethat a user gazes at a first device before issuing a voice command andgazes at a last device immediately after the voice command is issued.Meanwhile, the user may gazes at any device or devices when the user isissuing the command verbally. Then it may be configured that either thefirst device or the last device may dominate. With the first method, thecommand may be performed at the first device, regardless of what happensafterwards. With the second method, the command may be carried out atthe last device regardless of what happens before.

In above discussions, it is assumed that a device contains a gazesensor, a voice recognition system, and a presentation component like adisplay or a speaker. Alternatively, a device may only contain apresentation component and perform presentation function, while gazesensing and voice recognition may be controlled by a separate on-site orremote control system. For instance, a control system of a museum maymonitor a visitor's gaze direction and verbal instructions using gazingand voice sensors. The control system may detect whether the visitorlooks at a wall-mount display and says “Open” simultaneously or within agiven time period starting from end of the gazing act, or says “Open”and looks at the display simultaneously or within a given time periodstarting from end of the voice input submission. For instance, thecontrol system may receive and analyze data from the sensors, ascertaina visitor's gaze direction, identify the wall-mount display by the gazedirection, receive a voice input from the visitor, recognize a commandfrom the input by certain algorithm, determine time periodscorresponding to the gazing and verbal acts respectively, proceed whenthe two periods overlap or a gap between the two periods is smaller thana given value, generate a signal, and send out the signal which maycause the display to turn on and show certain contents accordingly.

A device may also have a locating detector to identify a user andmeasure position of the user who has just generated verbal contents. Alocating detector may measure and analyze sound waves to determine asource position using mature technologies. The locating detector mayalso be used to collect voice inputs from a target user only, where thetarget user may have gazed at a device or may be gazing at the device.Locating a target user becomes critical when multiple users are on site.For instance, a device may be configured to receive and interpret avoice input, identify and locate a user who just gives the voice inputusing a locating detector, measure the user's gazing direction, and thenperform a task extracted from the voice input when the user gazes at thedevice simultaneously or within a given time period after the voiceinput is received. Alternatively, a device may also be configured tomonitor a user's gaze direction, measure and obtain position data of theuser after the user gazes at the device, calculate a target position ofsound source of the user, e.g., a position of the user's head or mouth,receive a voice input, ascertain whether the input comes from the targetposition, analyze the input if it is from the target position, ascertainwhether the input contains a command, and then perform a task derivedfrom the command when the input is received while the user is stillgazing at the device or within a given time period after end of thegazing act.

It is noted that a user may generate a voice input which may includevarious simple or complex commands. A simple command may contain asingle and simple word to describe a simple task, such as “Start”,“Open”, or “TV”, which may be used to cause a device to start working,like turning on a radio, an air conditioning, or television. A user mayalso issue a complex command which may contain several sentences todescribe one or more tasks having several requirements. For instance auser may say to a control device “Turn on air conditioning, turn on TV,go to Channel Nine,” while looking at it.

Since a device may be targeted precisely with mature voice recognitiontechniques, gaze sensing may not be needed in some cases. For instance,a name may be assigned to a device. When a user says the name and acommand, the device may detect its name and take the command. Butrelying solely on a device name in a voice command has weaknesses. Forinstance, a user has to remember a name, which has to be unique to avoidduplicating another name. A user has to say the name, which means anextra requirement and extra step. And a user may say a wrong name, whichmay cause frustration since a command may not be carried out. Thus thereexists a need for a method which combines gaze sensing and voicerecognition via a new scheme.

When a name is assigned to a device, a voice command may be taken from auser and implemented at the device in several cases. For instance, adevice may monitor a user's gaze direction and voice input and carriedout a command when one of the conditions or requirements is satisfiedwithout using name of the device. The conditions or requirements may bethose which are discussed in the above, e.g., when a gazing and verbalacts occur together. A device may also be configured to recognize itsname from a voice input and implement a command without checking gazedirection. For instance, assume that a device is assigned a name “ABW”.The device's voice recognition sensor is on. After a user says “ABW,turn on the lights”, the device may take the input, recognize its nameand the command, and then create a signal to turn on the lights, whichis a task derived from the command. But if a wrong name is used, thedevice may not follow the command. A device may implement a command evenwhen a user says a wrong name if it relies on results of gaze detection.For instance, assume a user says to the device “YW, turn on the lights”while looking at it. If voice recognition is used alone, the device maynot react, as the command is addressed to another device. However, withgaze detection, it may be configured that as long as a user gazes at adevice while speaking to it, or a user's gazing and verbal acts satisfyone of the conditions or requirements, a command may be implemented evenwhen the user says a wrong name.

To make it more flexible, three options may be provided to a user at thesame time: A user may gaze at a device and say a command withoutmentioning a device name; a user may say a command and mention a correctdevice name without gazing at it; and a user may gaze at a device, say acommand, and mention a device name. The first option represents allcases as discussed where a device name is not involved. The secondoption may not work if a wrong name is used. The third option is likethe first option plus that a user says a device name. In the thirdoption, whether or not a user mentions a correct name becomesirrelevant, since the device is identified by detected gazing direction,not the device name. Therefore, a user may choose to gaze at a device ornot to gaze at it, when issuing a voice command to the device. And to becertain, a user may choose to gaze at a device when submitting a voicecommand.

Accordingly, a device may be configured for a user to use with any ofthe three options. For instance, a device may keep monitoring a user'svoice input and gaze direction via a voice recognition system and a gazesensor, and ascertain whether a voice input contains a command andwhether the user gazes at the device. If the device doesn't detect anycommand from the user, no task is implemented. If the device detects avoice command, it may ascertain whether a qualified gazing act happensand whether a device name is mentioned which matches a predeterminedname or profile. A qualified gazing act may be the one which whencombined with the verbal act satisfies one of aforementioned conditionsor requirements. If a qualified gazing act is detected, the devicestarts implementing the command. If a qualified gazing act is notdetected, but a correct device name is mentioned, the device startsimplementing the command. If a qualified gazing act is not detected, anda correct device name is not mentioned, the device doesn't implement thecommand.

When multiple devices are involved, an on-site or remote control systemmay be arranged. The control system may receive, collect, and analyzedata sent from gaze sensors and voice sensing detectors of the devices.A voice sensing detector may be designed to detect sound waves. The gazesensors and voice sensing detectors may be arranged to sense a usercontinuously. The control system may work in three modes. In the firstmode, the control system may carry out a command at a device which auser gazes at and a condition set forth for gazing and verbal acts ismet. In the second mode, the control system may carry out a command at adevice whose name is mentioned by a user in the command. In the thirdmode, the control system may carry out a command at a first device whichis gazed at by a user or mentioned in the command. When a user gazes atthe first device and mentions a name of a second device, the controlsystem may carry out the command either at the first device or thesecond device depending on set-up selection. It may be arranged that auser may choose a mode or switch from a mode to another one.

CONCLUSION, RAMIFICATIONS, AND SCOPE

Thus it can be seen that systems and methods are introduced to presentlocation related information.

The improved method and system have the following features andadvantages:

-   -   (1). Location related info may be brought out by simply gazing        at a screen;    -   (2). Gaze sensor of a device may be turned on by a        location-based signal or shaking act of a user;    -   (3). Gaze detection and voice recognition may be used together,        where a device may be locked in by gaze detection and multiple        tasks may be arranged for selection via voice instructions;    -   (4). Information may be sorted and selected by pointing        direction of a device.

Although the description above contains many specificities, these shouldnot be construed as limiting the scope of the invention but as merelyproviding illustrations of some of the presently preferred embodiments.Numerous modifications will be obvious to those skilled in the art.

Ramifications:

Ambient light sensor may be added to a device which may be used to senseambient light intensity to determine whether the device is in a pocketor bag. If a device is not pulled out, act of shaking, tapping orknocking may be ignored in applications discussed in the above.

Furthermore, a device may be equipped with facial recognition system.The system may at least recognize a device owner, which may protect userprivacy by not following other people's instructions. The system maymake use of eye-tracking camera and employ facial sensing algorithm toidentify a user.

In several examples in the above, shaking is used to illustrate variousembodiments. It is noted that other forms of user input, such asrotating, tilting, tapping, knocking, or touching may also be employedto perform the same function. Still other methods for a user to interactwith a device besides shaking include sliding on a touch screen or touchpad, or opening a lid of a device.

If a user's identity is known, info presented may be selected based onnot only the location of the user, but also his or her past experience,or based on user's experience only when it is worthwhile to do so. Forinstance, when a user is travelling, he or she may have more free time,which provides a good chance for presenting certain info, likecommercials. Although advertisement presented may be unrelated to alocation, since it may fit the interest or need of the user, it may berelevant, and thus effective. Thus, contents presented after waking upfrom standby state via gazing detection may be location related orarranged at a remote or nearby facility according to a user's recordsand analysis on the user.

In addition, it may be arranged such that a user may decide what topresent on screen. For instance, a user may choose a program for updateon news, instant messages, or a social networking group. When thereisn't any suitable location related info or other prearranged info froma service, a device may display contents preselected by a user after thedevice received a signal and appropriate user input. Options may also beconfigured such that a user may select either to present locationrelated info prepared by a service or present certain info or programprearranged by a user. In practice, buttons “Location-Related Info” and“Self-Selection” may be configured on screen during a presentationperiod. The buttons may correspond to presentation of location-basedinfo and self-selected info respectively. A user may switch betweenshowing different information by tapping or clicking on the buttons.

Thus, a gazing act may be used to display info which is related to auser's location, records, or self-selection. It is noted that info ondisplay or to be displayed may be any which is arranged by a system, aservice, or a user, or may be any which is not related to contents shownon screen before a device got into a standby mode. In other words, itmay be arranged such that a gazing act or shaking plus gazing act causespresentation of contents which are specifically arranged and areunrelated to information presented before standby state.

The schemes illustrated in FIGS. 2, 3, and 4 may be combined in manyways. For example, sorting info by pointing function may be applied toall cases. On-screen buttons may be arranged for turning on and off thesorting function. So a user may choose options to turn on or off sortingmode any time.

For convenience and better user experience, after a user arrives at aplace and user's device receives location related data, the device maybeep to signal that some info is available to watch. Meanwhile, if thedevice is standby, a small window may appear to post a short notice.Thus, a user may easily know there is something available and then maygaze at the screen or notice window to invoke a presentation or shakethe device and then gaze at it to cause info presentation.

For a qualified gaze or look at a display, a user's eye may also fall onthings located outside of the display but close to its edge, instead ofareas on display only. The reason is that, when a user looks at objectsclose to a display, contents shown on it may also reach the user's eye,thus providing a viewing opportunity anyway. And hopefully, the user mayturn his or her sight a bit to get a better reception of the contents.Moreover in many cases, instead of a display of a device, it may be goodenough to qualify as a gaze at a display if a user just looks at adirection toward the device, because it means a user may have intentionto watch a presentation on it, and the user may have a good chance tonotice contents displayed on the device anyway. In cases of smartphoneand tablet computer, gazing at a device is almost equivalent to gazingat a display, because for these devices, a display may cover the wholearea of one side.

Lastly, a method may be configured which ascertains whether a user facesa device, instead of gazing at a device. In some applications, it may bedifficult to sense a user's eye movement, due to technical issues orambient lighting conditions. Thus it may be arranged to detect whether auser faces a device. For instance, a device may use an imaging sensorlike camera to take pictures or videos of a user. Certain algorithm maybe used to identify facial features of the user, determine positions ofthe user's eyes, and then calculate a distance between a spot of thedevice and one eye and another distance between the spot and the othereye. The spot may be a point at the center of the device or the centerof an output component. If difference of the two distances is smallerthan a given value, it may be considered that the device is right infront of the user or the user faces the device. Consequently, it may beconfigured that in all of above discussions, gazing requirement may bereplaced by facing requirement when a user or entity decides to do so.For instance, a requirement of gazing at a device may become arequirement of facing a device.

Therefore the scope of the invention should be determined by theappended claims and their legal equivalents, rather than by the examplesgiven.

The invention claimed is:
 1. A method performed for presentinginformation using an electronic device, comprising: 1) arranging a voicerecognition sensor; 2) receiving a voice input from a user via saidvoice recognition sensor; 3) ascertaining whether said voice inputmatches a given profile; 4) configuring a gaze sensor; 5) sensing saiduser using said gaze sensor and determining whether said user gazes at adirection toward said electronic device; and 6) presenting a pluralityof contents at said electronic device when said voice input matches saidgiven profile and it is detected that said user gazes at a directiontoward said electronic device within a time period after said voiceinput is received.
 2. The method according to claim 1, wherein thesensing step is triggered when said voice input matches said givenprofile.
 3. The method according to claim 1, wherein said given profilecontains a single word.
 4. The method according to claim 1, wherein saidplurality of contents is presented when it is detected that said usergazes at a direction toward said electronic device for at least acertain time period after said voice input is received.
 5. A methodperformed for presenting information using an electronic device,comprising: 1) arranging a voice recognition sensor; 2) receiving avoice input from a user via said voice recognition sensor; 3)ascertaining whether said voice input matches a given profile; 4)configuring a gaze sensor; 5) starting sensing said user using said gazesensor when said voice input matches said given profile; 6) determiningwhether said user gazes at a direction toward said electronic device;and 7) presenting a plurality of contents at said electronic deviceafter it is detected that said user gazes at a direction toward saidelectronic device within a time period after said voice input isreceived.
 6. The method according to claim 5, wherein said given profilecontains a single word.
 7. The method according to claim 5, wherein saidplurality of contents is presented when it is detected that said usergazes at a direction toward said electronic device for at least acertain time period after said voice input is received.
 8. An electronicapparatus comprising: one or more processors; and one or more memorydevices coupled to said one or more processors, said one or moreprocessors operable when executing certain instructions to: 1) utilize avoice recognition sensor; 2) receive a voice input from a user via saidvoice recognition sensor; 3) ascertain whether said voice input matchesa given profile; 4) utilize a gaze sensor; 5) sense said user using saidgaze sensor and determine whether said user gazes at a direction towardsaid electronic apparatus; and 6) present a plurality of contents usingsaid electronic apparatus when said voice input matches said givenprofile and it is detected that said user gazes at a direction towardsaid electronic apparatus within a time period after said voice input isreceived.
 9. The apparatus according to claim 8, wherein the sensingstep is triggered when said voice input matches said given profile. 10.The apparatus according to claim 8, wherein said given profile containsa single word.
 11. The apparatus according to claim 8, wherein saidplurality of contents is presented when it is detected that said usergazes at a direction toward said electronic apparatus for at least acertain time period after said voice input is received.